Improving assessment evidence in e-learning products: some solutions for reliability

E-learning products such as cognitive diagnosers interact with learners and collect assessment data to build a picture of some aspect of a learner’s thinking. One concern for this rapidly emerging area of e-learning is whether the diagnostic conclusions of such products are based on sound evidence, including whether or not the diagnostics are reliable. In online settings, the information may be used for adaptive delivery of content, individualizing learning materials, dynamic feedback, teacher feed-forward, cognitive mapping, score reporting and course placement. A reliability index quantifies the impact that measurement error at the individual level may have on the accuracy of the inference. This paper investigates some simple solutions that substantially improve reliability within one e-learning product. These solutions include providing questions of appropriate difficulty that help to maximize item information across the distribution.